2026-04-02 09:50:42,300 [ 151463 ] INFO : ClickHouse root is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse (runner:42, check_args_and_update_paths) 2026-04-02 09:50:42,301 [ 151463 ] INFO : Cases dir is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration (runner:86, check_args_and_update_paths) 2026-04-02 09:50:42,301 [ 151463 ] INFO : utils dir is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse/utils (runner:97, check_args_and_update_paths) 2026-04-02 09:50:42,301 [ 151463 ] INFO : base_configs_dir: /home/ubuntu/_work/ClickHouse/ClickHouse/programs/server, binary: /home/ubuntu/_work/_temp/test/build/clickhouse, cases_dir: /home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration (runner:99, check_args_and_update_paths) clickhouse_integration_tests_volume Running pytest container as: 'docker run --rm --name clickhouse_integration_tests_v1snvh --privileged --dns-search='.' --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-odbc-bridge:/clickhouse-odbc-bridge --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-library-bridge:/clickhouse-library-bridge --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/backupview:/ClickHouse/utils/backupview --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/grpc-client/pb2:/ClickHouse/utils/grpc-client/pb2 --volume=/run:/run/host:ro --volume=clickhouse_integration_tests_volume:/var/lib/docker -e DOCKER_DOTNET_CLIENT_TAG=11de0b29a15d -e DOCKER_HELPER_TAG=2cffe1eae894 -e DOCKER_BASE_TAG=1e0b53d756cf -e DOCKER_KERBEROS_KDC_TAG=9391ecdee8d7 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=9bec2a638e6e -e DOCKER_MYSQL_JAVA_CLIENT_TAG=766bff31cfe4 -e DOCKER_MYSQL_JS_CLIENT_TAG=41ba7c2ec2a1 -e DOCKER_MYSQL_PHP_CLIENT_TAG=88be89c1e3b6 -e DOCKER_NGINX_DAV_TAG=b55ac9cd7519 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=a4eff5c7f4d6 -e DOCKER_PYTHON_BOTTLE_TAG=caad4729259e -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e CLICKHOUSE_USE_OLD_ANALYZER=1 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="--dist=loadfile -n 10 -rfEps --run-id=0 --color=no --durations=0 'test_distributed_format/test.py::test_single_file[test_cluster_internal_replication]' 'test_distributed_format/test.py::test_single_file[test_cluster_no_internal_replication]' 'test_distributed_format/test.py::test_single_file_old[test_cluster_internal_replication]' 'test_distributed_format/test.py::test_single_file_old[test_cluster_no_internal_replication]' 'test_distributed_format/test.py::test_two_files[test_cluster_internal_replication]' 'test_distributed_format/test.py::test_two_files[test_cluster_no_internal_replication]' test_distributed_insert_backward_compatibility/test.py::test_distributed_in_tuple test_distributed_inter_server_secret/test.py::test_insecure test_distributed_inter_server_secret/test.py::test_insecure_insert_async test_distributed_inter_server_secret/test.py::test_insecure_insert_sync 'test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_insecure_cluster[default-]' 'test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_insecure_cluster[nopass-]' 'test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_insecure_cluster[pass-foo]' 'test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_secure_cluster[default-]' 'test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_secure_cluster[nopass-]' 'test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_secure_cluster[pass-foo]' 'test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[default-]' 'test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[nopass-]' 'test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[pass-foo]' 'test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[default-]' 'test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[nopass-]' 'test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[pass-foo]' test_distributed_inter_server_secret/test.py::test_secure test_distributed_inter_server_secret/test.py::test_secure_cluster_distributed_over_distributed_different_users_cluster test_distributed_inter_server_secret/test.py::test_secure_cluster_distributed_over_distributed_different_users_remote test_distributed_inter_server_secret/test.py::test_secure_disagree test_distributed_inter_server_secret/test.py::test_secure_disagree_insert test_distributed_inter_server_secret/test.py::test_secure_insert_async test_distributed_inter_server_secret/test.py::test_secure_insert_buffer_async test_distributed_inter_server_secret/test.py::test_secure_insert_sync 'test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[default-]' 'test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[nopass-]' 'test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[pass-foo]' 'test_distributed_inter_server_secret/test.py::test_user_secure_cluster[default-]' 'test_distributed_inter_server_secret/test.py::test_user_secure_cluster[nopass-]' 'test_distributed_inter_server_secret/test.py::test_user_secure_cluster[pass-foo]' test_distributed_load_balancing/test.py::test_distributed_replica_max_ignored_errors test_distributed_load_balancing/test.py::test_load_balancing_default test_distributed_load_balancing/test.py::test_load_balancing_first_or_random test_distributed_load_balancing/test.py::test_load_balancing_hostname_levenshtein_distance test_distributed_load_balancing/test.py::test_load_balancing_in_order test_distributed_load_balancing/test.py::test_load_balancing_nearest_hostname 'test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority]' 'test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority_negative]' test_distributed_load_balancing/test.py::test_load_balancing_round_robin test_dotnet_client/test.py::test_dotnet_client test_encrypted_disk_replication/test.py::test_replication test_encrypted_disk_replication/test.py::test_zero_copy_replication test_endpoint_macro_substitution/test.py::test_different_types test_endpoint_macro_substitution/test.py::test_select_by_type test_executable_user_defined_functions_config_reload/test.py::test test_globs_in_filepath/test.py::test_deep_structure test_globs_in_filepath/test.py::test_linear_structure test_globs_in_filepath/test.py::test_strange_filenames test_globs_in_filepath/test.py::test_table_function_and_virtual_columns test_graphite_merge_tree/test.py::test_broken_partial_rollup test_graphite_merge_tree/test.py::test_combined_rules test_graphite_merge_tree/test.py::test_combined_rules_with_default test_graphite_merge_tree/test.py::test_multiple_output_blocks test_graphite_merge_tree/test.py::test_multiple_paths_and_versions test_graphite_merge_tree/test.py::test_path_dangling_pointer test_graphite_merge_tree/test.py::test_paths_not_matching_any_pattern test_graphite_merge_tree/test.py::test_rollup_aggregation test_graphite_merge_tree/test.py::test_rollup_aggregation_2 test_graphite_merge_tree/test.py::test_rollup_versions test_graphite_merge_tree/test.py::test_system_graphite_retentions test_graphite_merge_tree/test.py::test_wrong_rollup_config test_grpc_protocol_ssl/test.py::test_insecure_channel test_grpc_protocol_ssl/test.py::test_secure_channel test_grpc_protocol_ssl/test.py::test_wrong_client_certificate test_hedged_requests/test.py::test_async_connect test_hedged_requests/test.py::test_async_query_sending test_hedged_requests/test.py::test_combination1 test_hedged_requests/test.py::test_combination2 test_hedged_requests/test.py::test_combination3 test_hedged_requests/test.py::test_combination4 test_hedged_requests/test.py::test_initial_receive_timeout test_hedged_requests/test.py::test_long_query test_hedged_requests/test.py::test_receive_timeout1 test_hedged_requests/test.py::test_receive_timeout2 test_hedged_requests/test.py::test_send_data test_hedged_requests/test.py::test_send_data2 test_hedged_requests/test.py::test_send_table_status_sleep test_hedged_requests/test.py::test_send_table_status_sleep2 test_hedged_requests/test.py::test_stuck_replica test_hot_reload_storage_policy/test.py::test_hot_reload_policy test_hot_reload_storage_policy/test.py::test_hot_reload_policy_distributed_table test_http_handlers_config/test.py::test_absolute_path_static_handler test_http_handlers_config/test.py::test_config_static_handler test_http_handlers_config/test.py::test_defaults_http_handlers test_http_handlers_config/test.py::test_defaults_http_handlers_config_order test_http_handlers_config/test.py::test_dynamic_query_handler test_http_handlers_config/test.py::test_fixed_static_handler test_http_handlers_config/test.py::test_headers_in_response test_http_handlers_config/test.py::test_predefined_query_handler test_http_handlers_config/test.py::test_prometheus_handler test_http_handlers_config/test.py::test_relative_path_static_handler test_http_handlers_config/test.py::test_replicas_status_handler test_input_format_parallel_parsing_memory_tracking/test.py::test_memory_tracking_total 'test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_corrupted_big[0]' -vvv" altinityinfra/integration-tests-runner:37a9815fd2fa '. Start tests ============================= test session starts ============================== platform linux -- Python 3.10.12, pytest-7.4.4, pluggy-1.5.0 -- /usr/bin/python3 cachedir: .pytest_cache rootdir: /ClickHouse/tests/integration configfile: pytest.ini plugins: order-1.0.1, random-0.2, timeout-2.2.0, repeat-0.9.3, reportlog-0.4.0, xdist-3.5.0 timeout: 900.0s timeout method: signal timeout func_only: False created: 10/10 workers 10 workers [100 items] scheduling tests via LoadFileScheduling test_distributed_load_balancing/test.py::test_distributed_replica_max_ignored_errors test_distributed_inter_server_secret/test.py::test_insecure test_endpoint_macro_substitution/test.py::test_different_types test_hedged_requests/test.py::test_async_connect test_distributed_format/test.py::test_single_file[test_cluster_internal_replication] test_http_handlers_config/test.py::test_absolute_path_static_handler test_graphite_merge_tree/test.py::test_broken_partial_rollup test_globs_in_filepath/test.py::test_deep_structure test_grpc_protocol_ssl/test.py::test_insecure_channel test_encrypted_disk_replication/test.py::test_replication [gw5] [ 1%] PASSED test_grpc_protocol_ssl/test.py::test_insecure_channel test_grpc_protocol_ssl/test.py::test_secure_channel [gw5] [ 2%] PASSED test_grpc_protocol_ssl/test.py::test_secure_channel test_grpc_protocol_ssl/test.py::test_wrong_client_certificate [gw5] [ 3%] PASSED test_grpc_protocol_ssl/test.py::test_wrong_client_certificate [gw1] [ 4%] PASSED test_graphite_merge_tree/test.py::test_broken_partial_rollup [gw6] [ 5%] PASSED test_distributed_format/test.py::test_single_file[test_cluster_internal_replication] test_distributed_format/test.py::test_single_file[test_cluster_no_internal_replication] test_graphite_merge_tree/test.py::test_combined_rules [gw1] [ 6%] PASSED test_graphite_merge_tree/test.py::test_combined_rules test_graphite_merge_tree/test.py::test_combined_rules_with_default [gw6] [ 7%] PASSED test_distributed_format/test.py::test_single_file[test_cluster_no_internal_replication] test_distributed_format/test.py::test_single_file_old[test_cluster_internal_replication] [gw7] [ 8%] PASSED test_http_handlers_config/test.py::test_absolute_path_static_handler test_http_handlers_config/test.py::test_config_static_handler [gw1] [ 9%] PASSED test_graphite_merge_tree/test.py::test_combined_rules_with_default test_dotnet_client/test.py::test_dotnet_client test_graphite_merge_tree/test.py::test_multiple_output_blocks [gw6] [ 10%] PASSED test_distributed_format/test.py::test_single_file_old[test_cluster_internal_replication] test_distributed_format/test.py::test_single_file_old[test_cluster_no_internal_replication] [gw1] [ 11%] PASSED test_graphite_merge_tree/test.py::test_multiple_output_blocks test_graphite_merge_tree/test.py::test_multiple_paths_and_versions [gw6] [ 12%] PASSED test_distributed_format/test.py::test_single_file_old[test_cluster_no_internal_replication] test_distributed_format/test.py::test_two_files[test_cluster_internal_replication] [gw1] [ 13%] PASSED test_graphite_merge_tree/test.py::test_multiple_paths_and_versions test_graphite_merge_tree/test.py::test_path_dangling_pointer [gw6] [ 14%] PASSED test_distributed_format/test.py::test_two_files[test_cluster_internal_replication] test_distributed_format/test.py::test_two_files[test_cluster_no_internal_replication] [gw0] [ 15%] PASSED test_distributed_inter_server_secret/test.py::test_insecure test_distributed_inter_server_secret/test.py::test_insecure_insert_async [gw9] [ 16%] PASSED test_encrypted_disk_replication/test.py::test_replication [gw0] [ 17%] PASSED test_distributed_inter_server_secret/test.py::test_insecure_insert_async test_distributed_inter_server_secret/test.py::test_insecure_insert_sync test_encrypted_disk_replication/test.py::test_zero_copy_replication [gw2] [ 18%] FAILED test_distributed_load_balancing/test.py::test_distributed_replica_max_ignored_errors test_distributed_load_balancing/test.py::test_load_balancing_default [gw6] [ 19%] PASSED test_distributed_format/test.py::test_two_files[test_cluster_no_internal_replication] [gw0] [ 20%] PASSED test_distributed_inter_server_secret/test.py::test_insecure_insert_sync test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_insecure_cluster[default-] [gw9] [ 21%] PASSED test_encrypted_disk_replication/test.py::test_zero_copy_replication [gw0] [ 22%] PASSED test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_insecure_cluster[default-] test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_insecure_cluster[nopass-] [gw7] [ 23%] PASSED test_http_handlers_config/test.py::test_config_static_handler test_http_handlers_config/test.py::test_defaults_http_handlers test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_corrupted_big[0] [gw0] [ 24%] PASSED test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_insecure_cluster[nopass-] test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_insecure_cluster[pass-foo] [gw0] [ 25%] PASSED test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_insecure_cluster[pass-foo] test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_secure_cluster[default-] [gw5] [ 26%] PASSED test_dotnet_client/test.py::test_dotnet_client [gw0] [ 27%] PASSED test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_secure_cluster[default-] test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_secure_cluster[nopass-] [gw4] [ 28%] PASSED test_hedged_requests/test.py::test_async_connect test_hedged_requests/test.py::test_async_query_sending [gw0] [ 29%] PASSED test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_secure_cluster[nopass-] test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_secure_cluster[pass-foo] [gw0] [ 30%] PASSED test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_secure_cluster[pass-foo] test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[default-] [gw8] [ 31%] PASSED test_endpoint_macro_substitution/test.py::test_different_types test_endpoint_macro_substitution/test.py::test_select_by_type test_executable_user_defined_functions_config_reload/test.py::test [gw8] [ 32%] PASSED test_endpoint_macro_substitution/test.py::test_select_by_type [gw0] [ 33%] PASSED test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[default-] test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[nopass-] [gw1] [ 34%] PASSED test_graphite_merge_tree/test.py::test_path_dangling_pointer test_graphite_merge_tree/test.py::test_paths_not_matching_any_pattern test_hot_reload_storage_policy/test.py::test_hot_reload_policy [gw1] [ 35%] PASSED test_graphite_merge_tree/test.py::test_paths_not_matching_any_pattern [gw0] [ 36%] PASSED test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[nopass-] test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[pass-foo] test_graphite_merge_tree/test.py::test_rollup_aggregation [gw0] [ 37%] PASSED test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[pass-foo] test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[default-] [gw1] [ 38%] PASSED test_graphite_merge_tree/test.py::test_rollup_aggregation test_graphite_merge_tree/test.py::test_rollup_aggregation_2 [gw1] [ 39%] PASSED test_graphite_merge_tree/test.py::test_rollup_aggregation_2 test_graphite_merge_tree/test.py::test_rollup_versions [gw7] [ 40%] PASSED test_http_handlers_config/test.py::test_defaults_http_handlers test_http_handlers_config/test.py::test_defaults_http_handlers_config_order [gw0] [ 41%] PASSED test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[default-] test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[nopass-] [gw1] [ 42%] PASSED test_graphite_merge_tree/test.py::test_rollup_versions test_graphite_merge_tree/test.py::test_system_graphite_retentions [gw0] [ 43%] PASSED test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[nopass-] test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[pass-foo] [gw1] [ 44%] PASSED test_graphite_merge_tree/test.py::test_system_graphite_retentions test_graphite_merge_tree/test.py::test_wrong_rollup_config test_distributed_insert_backward_compatibility/test.py::test_distributed_in_tuple [gw0] [ 45%] PASSED test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[pass-foo] test_distributed_inter_server_secret/test.py::test_secure [gw6] [ 46%] PASSED test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_corrupted_big[0] [gw0] [ 47%] PASSED test_distributed_inter_server_secret/test.py::test_secure test_distributed_inter_server_secret/test.py::test_secure_cluster_distributed_over_distributed_different_users_cluster [gw1] [ 48%] PASSED test_graphite_merge_tree/test.py::test_wrong_rollup_config [gw9] [ 49%] PASSED test_hot_reload_storage_policy/test.py::test_hot_reload_policy test_hot_reload_storage_policy/test.py::test_hot_reload_policy_distributed_table [gw0] [ 50%] PASSED test_distributed_inter_server_secret/test.py::test_secure_cluster_distributed_over_distributed_different_users_cluster test_distributed_inter_server_secret/test.py::test_secure_cluster_distributed_over_distributed_different_users_remote [gw0] [ 51%] PASSED test_distributed_inter_server_secret/test.py::test_secure_cluster_distributed_over_distributed_different_users_remote test_distributed_inter_server_secret/test.py::test_secure_disagree [gw0] [ 52%] PASSED test_distributed_inter_server_secret/test.py::test_secure_disagree test_distributed_inter_server_secret/test.py::test_secure_disagree_insert [gw0] [ 53%] PASSED test_distributed_inter_server_secret/test.py::test_secure_disagree_insert test_distributed_inter_server_secret/test.py::test_secure_insert_async [gw0] [ 54%] PASSED test_distributed_inter_server_secret/test.py::test_secure_insert_async test_distributed_inter_server_secret/test.py::test_secure_insert_buffer_async [gw0] [ 55%] PASSED test_distributed_inter_server_secret/test.py::test_secure_insert_buffer_async test_distributed_inter_server_secret/test.py::test_secure_insert_sync [gw4] [ 56%] PASSED test_hedged_requests/test.py::test_async_query_sending test_hedged_requests/test.py::test_combination1 [gw5] [ 57%] PASSED test_executable_user_defined_functions_config_reload/test.py::test [gw0] [ 58%] PASSED test_distributed_inter_server_secret/test.py::test_secure_insert_sync test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[default-] [gw9] [ 59%] PASSED test_hot_reload_storage_policy/test.py::test_hot_reload_policy_distributed_table [gw0] [ 60%] PASSED test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[default-] test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[nopass-] test_input_format_parallel_parsing_memory_tracking/test.py::test_memory_tracking_total [gw0] [ 61%] PASSED test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[nopass-] test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[pass-foo] [gw0] [ 62%] PASSED test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[pass-foo] test_distributed_inter_server_secret/test.py::test_user_secure_cluster[default-] [gw7] [ 63%] PASSED test_http_handlers_config/test.py::test_defaults_http_handlers_config_order test_http_handlers_config/test.py::test_dynamic_query_handler [gw8] [ 64%] PASSED test_distributed_insert_backward_compatibility/test.py::test_distributed_in_tuple [gw0] [ 65%] PASSED test_distributed_inter_server_secret/test.py::test_user_secure_cluster[default-] test_distributed_inter_server_secret/test.py::test_user_secure_cluster[nopass-] [gw4] [ 66%] PASSED test_hedged_requests/test.py::test_combination1 test_hedged_requests/test.py::test_combination2 [gw0] [ 67%] PASSED test_distributed_inter_server_secret/test.py::test_user_secure_cluster[nopass-] test_distributed_inter_server_secret/test.py::test_user_secure_cluster[pass-foo] [gw0] [ 68%] PASSED test_distributed_inter_server_secret/test.py::test_user_secure_cluster[pass-foo] [gw5] [ 69%] PASSED test_input_format_parallel_parsing_memory_tracking/test.py::test_memory_tracking_total [gw7] [ 70%] PASSED test_http_handlers_config/test.py::test_dynamic_query_handler test_http_handlers_config/test.py::test_fixed_static_handler [gw4] [ 71%] PASSED test_hedged_requests/test.py::test_combination2 test_hedged_requests/test.py::test_combination3 [gw7] [ 72%] PASSED test_http_handlers_config/test.py::test_fixed_static_handler test_http_handlers_config/test.py::test_headers_in_response [gw4] [ 73%] PASSED test_hedged_requests/test.py::test_combination3 test_hedged_requests/test.py::test_combination4 [gw7] [ 74%] PASSED test_http_handlers_config/test.py::test_headers_in_response test_http_handlers_config/test.py::test_predefined_query_handler [gw4] [ 75%] PASSED test_hedged_requests/test.py::test_combination4 test_hedged_requests/test.py::test_initial_receive_timeout [gw7] [ 76%] PASSED test_http_handlers_config/test.py::test_predefined_query_handler test_http_handlers_config/test.py::test_prometheus_handler [gw2] [ 77%] PASSED test_distributed_load_balancing/test.py::test_load_balancing_default test_distributed_load_balancing/test.py::test_load_balancing_first_or_random [gw4] [ 78%] PASSED test_hedged_requests/test.py::test_initial_receive_timeout test_hedged_requests/test.py::test_long_query [gw7] [ 79%] PASSED test_http_handlers_config/test.py::test_prometheus_handler test_http_handlers_config/test.py::test_relative_path_static_handler [gw4] [ 80%] PASSED test_hedged_requests/test.py::test_long_query test_hedged_requests/test.py::test_receive_timeout1 [gw7] [ 81%] PASSED test_http_handlers_config/test.py::test_relative_path_static_handler test_http_handlers_config/test.py::test_replicas_status_handler [gw7] [ 82%] PASSED test_http_handlers_config/test.py::test_replicas_status_handler [gw4] [ 83%] PASSED test_hedged_requests/test.py::test_receive_timeout1 test_hedged_requests/test.py::test_receive_timeout2 [gw4] [ 84%] PASSED test_hedged_requests/test.py::test_receive_timeout2 test_hedged_requests/test.py::test_send_data [gw4] [ 85%] PASSED test_hedged_requests/test.py::test_send_data test_hedged_requests/test.py::test_send_data2 [gw3] [ 86%] PASSED test_globs_in_filepath/test.py::test_deep_structure test_globs_in_filepath/test.py::test_linear_structure [gw4] [ 87%] PASSED test_hedged_requests/test.py::test_send_data2 test_hedged_requests/test.py::test_send_table_status_sleep [gw3] [ 88%] PASSED test_globs_in_filepath/test.py::test_linear_structure test_globs_in_filepath/test.py::test_strange_filenames [gw3] [ 89%] PASSED test_globs_in_filepath/test.py::test_strange_filenames test_globs_in_filepath/test.py::test_table_function_and_virtual_columns [gw3] [ 90%] PASSED test_globs_in_filepath/test.py::test_table_function_and_virtual_columns [gw4] [ 91%] PASSED test_hedged_requests/test.py::test_send_table_status_sleep test_hedged_requests/test.py::test_send_table_status_sleep2 [gw4] [ 92%] PASSED test_hedged_requests/test.py::test_send_table_status_sleep2 test_hedged_requests/test.py::test_stuck_replica [gw2] [ 93%] PASSED test_distributed_load_balancing/test.py::test_load_balancing_first_or_random test_distributed_load_balancing/test.py::test_load_balancing_hostname_levenshtein_distance [gw4] [ 94%] PASSED test_hedged_requests/test.py::test_stuck_replica [gw2] [ 95%] PASSED test_distributed_load_balancing/test.py::test_load_balancing_hostname_levenshtein_distance test_distributed_load_balancing/test.py::test_load_balancing_in_order [gw2] [ 96%] PASSED test_distributed_load_balancing/test.py::test_load_balancing_in_order test_distributed_load_balancing/test.py::test_load_balancing_nearest_hostname [gw2] [ 97%] PASSED test_distributed_load_balancing/test.py::test_load_balancing_nearest_hostname test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority] [gw2] [ 98%] PASSED test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority] test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority_negative] [gw2] [ 99%] PASSED test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority_negative] test_distributed_load_balancing/test.py::test_load_balancing_round_robin [gw2] [100%] PASSED test_distributed_load_balancing/test.py::test_load_balancing_round_robin =================================== FAILURES =================================== _________________ test_distributed_replica_max_ignored_errors __________________ [gw2] linux -- Python 3.10.12 /usr/bin/python3 def test_distributed_replica_max_ignored_errors(): settings = { "use_hedged_requests": 0, "load_balancing": "in_order", "prefer_localhost_replica": 0, "connect_timeout": 2, "receive_timeout": 2, "send_timeout": 2, "idle_connection_timeout": 2, "tcp_keep_alive_timeout": 2, "distributed_replica_max_ignored_errors": 0, "distributed_replica_error_half_life": 60, } # initiate connection (if started only this test) n2.query("SELECT * FROM dist", settings=settings) cluster.pause_container("n1") # n1 paused -- skipping, and increment error_count for n1 # but the query succeeds, no need in query_and_get_error() n2.query("SELECT * FROM dist", settings=settings) # XXX: due to config reloading we need second time (sigh) n2.query("SELECT * FROM dist", settings=settings) # check error_count for n1 assert ( int( n2.query( """ SELECT errors_count FROM system.clusters WHERE cluster = 'replicas_cluster' AND host_name = 'n1' """, settings=settings, ) ) == 1 ) cluster.unpause_container("n1") # still n2 > assert get_node(n2, settings=settings) == "n2" E AssertionError: assert 'n3' == 'n2' E - n2 E + n3 test_distributed_load_balancing/test.py:234: AssertionError ---------------------------- Captured stdout setup ----------------------------- Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml ------------------------------ Captured log setup ------------------------------ 2026-04-02 09:50:54 [ 363 ] DEBUG : Command:['docker ps | wc -l'] (cluster.py:113, run_and_check) 2026-04-02 09:50:54 [ 363 ] DEBUG : Stdout:1 (cluster.py:121, run_and_check) 2026-04-02 09:50:54 [ 363 ] DEBUG : No running containers (conftest.py:95, cleanup_environment) 2026-04-02 09:50:54 [ 363 ] DEBUG : Pruning Docker networks (conftest.py:97, cleanup_environment) 2026-04-02 09:50:54 [ 363 ] DEBUG : Command:['docker network prune --force'] (cluster.py:113, run_and_check) 2026-04-02 09:50:54 [ 363 ] DEBUG : Stderr:Error response from daemon: a prune operation is already running (cluster.py:123, run_and_check) 2026-04-02 09:50:54 [ 363 ] DEBUG : Exitcode:1 (cluster.py:125, run_and_check) 2026-04-02 09:50:54 [ 363 ] DEBUG : Command:["sysctl net.ipv4.ip_local_port_range='55000 65535'"] (cluster.py:113, run_and_check) 2026-04-02 09:50:54 [ 363 ] DEBUG : Stdout:net.ipv4.ip_local_port_range = 55000 65535 (cluster.py:121, run_and_check) 2026-04-02 09:50:54 [ 363 ] INFO : Running tests in /ClickHouse/tests/integration/test_distributed_load_balancing/test.py (cluster.py:2788, start) 2026-04-02 09:50:54 [ 363 ] DEBUG : Cluster start called. is_up=False (cluster.py:2795, start) 2026-04-02 09:50:54 [ 363 ] DEBUG : Docker networks for project roottestdistributedloadbalancing_gw2 are NETWORK ID NAME DRIVER SCOPE (cluster.py:855, print_all_docker_pieces) 2026-04-02 09:50:54 [ 363 ] DEBUG : Docker containers for project roottestdistributedloadbalancing_gw2 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:863, print_all_docker_pieces) 2026-04-02 09:50:55 [ 363 ] DEBUG : Docker volumes for project roottestdistributedloadbalancing_gw2 are DRIVER VOLUME NAME (cluster.py:871, print_all_docker_pieces) 2026-04-02 09:50:55 [ 363 ] DEBUG : Cleanup called (cluster.py:876, cleanup) 2026-04-02 09:50:55 [ 363 ] DEBUG : Docker networks for project roottestdistributedloadbalancing_gw2 are NETWORK ID NAME DRIVER SCOPE (cluster.py:855, print_all_docker_pieces) 2026-04-02 09:50:55 [ 363 ] DEBUG : Docker containers for project roottestdistributedloadbalancing_gw2 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:863, print_all_docker_pieces) 2026-04-02 09:50:55 [ 363 ] DEBUG : Docker volumes for project roottestdistributedloadbalancing_gw2 are DRIVER VOLUME NAME (cluster.py:871, print_all_docker_pieces) 2026-04-02 09:50:55 [ 363 ] DEBUG : Command:docker container list --all --filter name='^/roottestdistributedloadbalancing_gw2_.*_1$' --format '{{.ID}}:{{.Names}}' (cluster.py:113, run_and_check) 2026-04-02 09:50:55 [ 363 ] DEBUG : Unstopped containers: {} (cluster.py:890, cleanup) 2026-04-02 09:50:55 [ 363 ] DEBUG : No running containers for project: roottestdistributedloadbalancing_gw2 (cluster.py:904, cleanup) 2026-04-02 09:50:55 [ 363 ] DEBUG : Trying to prune unused networks... (cluster.py:910, cleanup) 2026-04-02 09:50:55 [ 363 ] DEBUG : Trying to prune unused images... (cluster.py:926, cleanup) 2026-04-02 09:50:55 [ 363 ] DEBUG : Command:['docker', 'image', 'prune', '-f'] (cluster.py:113, run_and_check) 2026-04-02 09:50:55 [ 363 ] DEBUG : Stderr:Error response from daemon: a prune operation is already running (cluster.py:123, run_and_check) 2026-04-02 09:50:55 [ 363 ] DEBUG : Exitcode:1 (cluster.py:125, run_and_check) 2026-04-02 09:50:55 [ 363 ] DEBUG : Trying to prune unused volumes... (cluster.py:935, cleanup) 2026-04-02 09:50:55 [ 363 ] DEBUG : Command:['docker volume ls | wc -l'] (cluster.py:113, run_and_check) 2026-04-02 09:50:56 [ 363 ] DEBUG : Stdout:1 (cluster.py:121, run_and_check) 2026-04-02 09:50:56 [ 363 ] DEBUG : Setup directory for instance: n1 (cluster.py:2808, start) 2026-04-02 09:50:56 [ 363 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4534, create_dir) 2026-04-02 09:50:56 [ 363 ] DEBUG : Create directory for common tests configuration (cluster.py:4539, create_dir) 2026-04-02 09:50:56 [ 363 ] DEBUG : Copy common configuration from helpers (cluster.py:4559, create_dir) 2026-04-02 09:50:56 [ 363 ] DEBUG : Generate and write macros file (cluster.py:4602, create_dir) 2026-04-02 09:50:56 [ 363 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_distributed_load_balancing/configs/remote_servers.xml'] to /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0_gw2/n1/configs/config.d (cluster.py:4632, create_dir) 2026-04-02 09:50:56 [ 363 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0_gw2/n1/database (cluster.py:4649, create_dir) 2026-04-02 09:50:56 [ 363 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0_gw2/n1/logs (cluster.py:4660, create_dir) 2026-04-02 09:50:56 [ 363 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log"] (cluster.py:4746, create_dir) 2026-04-02 09:50:56 [ 363 ] DEBUG : Setup directory for instance: n2 (cluster.py:2808, start) 2026-04-02 09:50:56 [ 363 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4534, create_dir) 2026-04-02 09:50:56 [ 363 ] DEBUG : Create directory for common tests configuration (cluster.py:4539, create_dir) 2026-04-02 09:50:56 [ 363 ] DEBUG : Copy common configuration from helpers (cluster.py:4559, create_dir) 2026-04-02 09:50:56 [ 363 ] DEBUG : Generate and write macros file (cluster.py:4602, create_dir) 2026-04-02 09:50:56 [ 363 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_distributed_load_balancing/configs/remote_servers.xml'] to /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0_gw2/n2/configs/config.d (cluster.py:4632, create_dir) 2026-04-02 09:50:56 [ 363 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0_gw2/n2/database (cluster.py:4649, create_dir) 2026-04-02 09:50:56 [ 363 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0_gw2/n2/logs (cluster.py:4660, create_dir) 2026-04-02 09:50:56 [ 363 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log"] (cluster.py:4746, create_dir) 2026-04-02 09:50:56 [ 363 ] DEBUG : Setup directory for instance: n3 (cluster.py:2808, start) 2026-04-02 09:50:56 [ 363 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4534, create_dir) 2026-04-02 09:50:56 [ 363 ] DEBUG : Create directory for common tests configuration (cluster.py:4539, create_dir) 2026-04-02 09:50:56 [ 363 ] DEBUG : Copy common configuration from helpers (cluster.py:4559, create_dir) 2026-04-02 09:50:56 [ 363 ] DEBUG : Generate and write macros file (cluster.py:4602, create_dir) 2026-04-02 09:50:56 [ 363 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_distributed_load_balancing/configs/remote_servers.xml'] to /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0_gw2/n3/configs/config.d (cluster.py:4632, create_dir) 2026-04-02 09:50:56 [ 363 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0_gw2/n3/database (cluster.py:4649, create_dir) 2026-04-02 09:50:56 [ 363 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0_gw2/n3/logs (cluster.py:4660, create_dir) 2026-04-02 09:50:56 [ 363 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log"] (cluster.py:4746, create_dir) 2026-04-02 09:50:56 [ 363 ] DEBUG : Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw'} stored in /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0_gw2/.env (cluster.py:86, _create_env_file) 2026-04-02 09:50:56 [ 363 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file) 2026-04-02 09:50:56 [ 363 ] DEBUG : No config file found (config.py:28, find_config_file) 2026-04-02 09:50:56 [ 363 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file) 2026-04-02 09:50:56 [ 363 ] DEBUG : No config file found (config.py:28, find_config_file) 2026-04-02 09:50:56 [ 363 ] DEBUG : http://localhost:None "GET /version HTTP/1.1" 200 824 (connectionpool.py:547, _make_request) 2026-04-02 09:50:56 [ 363 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0_gw2/.env', '--project-name', 'roottestdistributedloadbalancing_gw2', '--file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0_gw2/n1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0_gw2/n2/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0_gw2/n3/docker-compose.yml', 'pull'] (cluster.py:113, run_and_check) 2026-04-02 09:51:09 [ 363 ] DEBUG : Stderr:Pulling n2 ... (cluster.py:123, run_and_check) 2026-04-02 09:51:09 [ 363 ] DEBUG : Stderr:Pulling n1 ... (cluster.py:123, run_and_check) 2026-04-02 09:51:09 [ 363 ] DEBUG : Stderr:Pulling n3 ... (cluster.py:123, run_and_check) 2026-04-02 09:51:09 [ 363 ] DEBUG : Stderr:Pulling n3 ... pulling from altinityinfra/integr... (cluster.py:123, run_and_check) 2026-04-02 09:51:09 [ 363 ] DEBUG : Stderr:Pulling n2 ... pulling from altinityinfra/integr... (cluster.py:123, run_and_check) 2026-04-02 09:51:09 [ 363 ] DEBUG : Stderr:Pulling n1 ... pulling from altinityinfra/integr... (cluster.py:123, run_and_check) 2026-04-02 09:51:09 [ 363 ] DEBUG : Stderr:Pulling n3 ... digest: sha256:bf725030a292d5daab... (cluster.py:123, run_and_check) 2026-04-02 09:51:09 [ 363 ] DEBUG : Stderr:Pulling n3 ... status: image is up to date for a... (cluster.py:123, run_and_check) 2026-04-02 09:51:09 [ 363 ] DEBUG : Stderr:Pulling n3 ... done (cluster.py:123, run_and_check) 2026-04-02 09:51:09 [ 363 ] DEBUG : Stderr:Pulling n2 ... digest: sha256:bf725030a292d5daab... (cluster.py:123, run_and_check) 2026-04-02 09:51:09 [ 363 ] DEBUG : Stderr:Pulling n2 ... status: image is up to date for a... (cluster.py:123, run_and_check) 2026-04-02 09:51:09 [ 363 ] DEBUG : Stderr:Pulling n2 ... done (cluster.py:123, run_and_check) 2026-04-02 09:51:09 [ 363 ] DEBUG : Stderr:Pulling n1 ... digest: sha256:bf725030a292d5daab... (cluster.py:123, run_and_check) 2026-04-02 09:51:09 [ 363 ] DEBUG : Stderr:Pulling n1 ... status: image is up to date for a... (cluster.py:123, run_and_check) 2026-04-02 09:51:09 [ 363 ] DEBUG : Stderr:Pulling n1 ... done (cluster.py:123, run_and_check) 2026-04-02 09:51:09 [ 363 ] DEBUG : ('Trying to create ClickHouse instance by command %s', 'docker-compose --env-file /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0_gw2/.env --project-name roottestdistributedloadbalancing_gw2 --file /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0_gw2/n1/docker-compose.yml --file /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0_gw2/n2/docker-compose.yml --file /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0_gw2/n3/docker-compose.yml up -d --no-recreate') (cluster.py:3146, start) 2026-04-02 09:51:09 [ 363 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0_gw2/.env', '--project-name', 'roottestdistributedloadbalancing_gw2', '--file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0_gw2/n1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0_gw2/n2/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0_gw2/n3/docker-compose.yml', 'up', '-d', '--no-recreate'] (cluster.py:113, run_and_check) 2026-04-02 09:51:17 [ 363 ] DEBUG : Stderr:Creating network "roottestdistributedloadbalancing_gw2_default" with the default driver (cluster.py:123, run_and_check) 2026-04-02 09:51:17 [ 363 ] DEBUG : Stderr:Creating roottestdistributedloadbalancing_gw2_n2_1 ... (cluster.py:123, run_and_check) 2026-04-02 09:51:17 [ 363 ] DEBUG : Stderr:Creating roottestdistributedloadbalancing_gw2_n3_1 ... (cluster.py:123, run_and_check) 2026-04-02 09:51:17 [ 363 ] DEBUG : Stderr:Creating roottestdistributedloadbalancing_gw2_n1_1 ... (cluster.py:123, run_and_check) 2026-04-02 09:51:17 [ 363 ] DEBUG : Stderr:Creating roottestdistributedloadbalancing_gw2_n1_1 ... done (cluster.py:123, run_and_check) 2026-04-02 09:51:17 [ 363 ] DEBUG : Stderr:Creating roottestdistributedloadbalancing_gw2_n3_1 ... done (cluster.py:123, run_and_check) 2026-04-02 09:51:17 [ 363 ] DEBUG : Stderr:Creating roottestdistributedloadbalancing_gw2_n2_1 ... done (cluster.py:123, run_and_check) 2026-04-02 09:51:17 [ 363 ] DEBUG : ClickHouse instance created (cluster.py:3154, start) 2026-04-02 09:51:17 [ 363 ] DEBUG : get_instance_ip instance_name=n1 (cluster.py:2135, get_instance_ip) 2026-04-02 09:51:17 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestdistributedloadbalancing_gw2_n1_1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:17 [ 363 ] DEBUG : Waiting for ClickHouse start in n1, ip: 172.16.5.2... (cluster.py:3161, start) 2026-04-02 09:51:17 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestdistributedloadbalancing_gw2_n1_1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:17 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:17 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:17 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:18 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:18 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:18 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:18 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:18 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:18 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:18 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:18 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:19 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:19 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:19 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:19 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:19 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:19 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:19 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:19 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:20 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:20 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:20 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:20 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:20 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:20 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:20 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:20 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:21 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:21 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:21 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:21 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:21 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:21 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:21 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:21 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:21 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:22 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:22 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:22 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:22 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:22 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:22 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:22 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:23 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:23 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:23 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:23 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:23 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:23 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:23 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:23 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:23 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:24 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:24 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:24 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:24 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:24 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:24 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:24 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:25 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:25 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:25 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:25 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:25 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:25 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:25 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/dbef7c46e0c59fe396c365c8d4f7b0dcc025df5c1544ac35c7b8155501cf7a41/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:25 [ 363 ] DEBUG : ClickHouse n1 started (cluster.py:3165, start) 2026-04-02 09:51:25 [ 363 ] DEBUG : get_instance_ip instance_name=n2 (cluster.py:2135, get_instance_ip) 2026-04-02 09:51:25 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestdistributedloadbalancing_gw2_n2_1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:25 [ 363 ] DEBUG : Waiting for ClickHouse start in n2, ip: 172.16.5.4... (cluster.py:3161, start) 2026-04-02 09:51:26 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestdistributedloadbalancing_gw2_n2_1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:26 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/05ceaa4d8f900d22949dffa2f05feaeaf0c91c446e5b9de7eab3fe4c2b5e784f/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:26 [ 363 ] DEBUG : ClickHouse n2 started (cluster.py:3165, start) 2026-04-02 09:51:26 [ 363 ] DEBUG : get_instance_ip instance_name=n3 (cluster.py:2135, get_instance_ip) 2026-04-02 09:51:26 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestdistributedloadbalancing_gw2_n3_1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:26 [ 363 ] DEBUG : Waiting for ClickHouse start in n3, ip: 172.16.5.3... (cluster.py:3161, start) 2026-04-02 09:51:26 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestdistributedloadbalancing_gw2_n3_1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:26 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/7acde7feef289bf2d125717ee1da33a1f428257aa3038bd4f7a4de3300a01d69/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:26 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/7acde7feef289bf2d125717ee1da33a1f428257aa3038bd4f7a4de3300a01d69/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:26 [ 363 ] DEBUG : http://localhost:None "GET /v1.42/containers/7acde7feef289bf2d125717ee1da33a1f428257aa3038bd4f7a4de3300a01d69/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 09:51:26 [ 363 ] DEBUG : ClickHouse n3 started (cluster.py:3165, start) 2026-04-02 09:51:26 [ 363 ] DEBUG : Executing query DROP TABLE IF EXISTS data on n1 (cluster.py:3602, query) 2026-04-02 09:51:26 [ 363 ] DEBUG : Executing query DROP TABLE IF EXISTS dist on n1 (cluster.py:3602, query) 2026-04-02 09:51:27 [ 363 ] DEBUG : Executing query CREATE TABLE data (key Int) Engine=Memory() on n1 (cluster.py:3602, query) 2026-04-02 09:51:28 [ 363 ] DEBUG : Executing query CREATE TABLE dist AS data Engine=Distributed( replicas_cluster, currentDatabase(), data) on n1 (cluster.py:3602, query) 2026-04-02 09:51:29 [ 363 ] DEBUG : Executing query CREATE TABLE dist_priority AS data Engine=Distributed( replicas_priority_cluster, currentDatabase(), data) on n1 (cluster.py:3602, query) 2026-04-02 09:51:30 [ 363 ] DEBUG : Executing query CREATE TABLE dist_priority_negative AS data Engine=Distributed( replicas_priority_negative_cluster, currentDatabase(), data) on n1 (cluster.py:3602, query) 2026-04-02 09:51:31 [ 363 ] DEBUG : Executing query DROP TABLE IF EXISTS data on n2 (cluster.py:3602, query) 2026-04-02 09:51:32 [ 363 ] DEBUG : Executing query DROP TABLE IF EXISTS dist on n2 (cluster.py:3602, query) 2026-04-02 09:51:34 [ 363 ] DEBUG : Executing query CREATE TABLE data (key Int) Engine=Memory() on n2 (cluster.py:3602, query) 2026-04-02 09:51:34 [ 363 ] DEBUG : Executing query CREATE TABLE dist AS data Engine=Distributed( replicas_cluster, currentDatabase(), data) on n2 (cluster.py:3602, query) 2026-04-02 09:51:35 [ 363 ] DEBUG : Executing query CREATE TABLE dist_priority AS data Engine=Distributed( replicas_priority_cluster, currentDatabase(), data) on n2 (cluster.py:3602, query) 2026-04-02 09:51:36 [ 363 ] DEBUG : Executing query CREATE TABLE dist_priority_negative AS data Engine=Distributed( replicas_priority_negative_cluster, currentDatabase(), data) on n2 (cluster.py:3602, query) 2026-04-02 09:51:38 [ 363 ] DEBUG : Executing query DROP TABLE IF EXISTS data on n3 (cluster.py:3602, query) 2026-04-02 09:51:39 [ 363 ] DEBUG : Executing query DROP TABLE IF EXISTS dist on n3 (cluster.py:3602, query) 2026-04-02 09:51:41 [ 363 ] DEBUG : Executing query CREATE TABLE data (key Int) Engine=Memory() on n3 (cluster.py:3602, query) 2026-04-02 09:51:42 [ 363 ] DEBUG : Executing query CREATE TABLE dist AS data Engine=Distributed( replicas_cluster, currentDatabase(), data) on n3 (cluster.py:3602, query) 2026-04-02 09:51:45 [ 363 ] DEBUG : Executing query CREATE TABLE dist_priority AS data Engine=Distributed( replicas_priority_cluster, currentDatabase(), data) on n3 (cluster.py:3602, query) 2026-04-02 09:51:46 [ 363 ] DEBUG : Executing query CREATE TABLE dist_priority_negative AS data Engine=Distributed( replicas_priority_negative_cluster, currentDatabase(), data) on n3 (cluster.py:3602, query) 2026-04-02 09:51:47 [ 363 ] DEBUG : Executing query SYSTEM RELOAD CONFIG on n1 (cluster.py:3602, query) 2026-04-02 09:51:49 [ 363 ] DEBUG : Executing query SYSTEM RELOAD CONFIG on n2 (cluster.py:3602, query) 2026-04-02 09:51:51 [ 363 ] DEBUG : Executing query SYSTEM RELOAD CONFIG on n3 (cluster.py:3602, query) ------------------------------ Captured log call ------------------------------- 2026-04-02 09:51:52 [ 363 ] DEBUG : Executing query SELECT * FROM dist on n2 (cluster.py:3602, query) 2026-04-02 09:51:54 [ 363 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0_gw2/.env', '--project-name', 'roottestdistributedloadbalancing_gw2', '--file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0_gw2/n1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0_gw2/n2/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0_gw2/n3/docker-compose.yml', 'pause', 'n1'] (cluster.py:113, run_and_check) 2026-04-02 09:51:57 [ 363 ] DEBUG : Stderr:Pausing roottestdistributedloadbalancing_gw2_n1_1 ... (cluster.py:123, run_and_check) 2026-04-02 09:51:57 [ 363 ] DEBUG : Stderr:Pausing roottestdistributedloadbalancing_gw2_n1_1 ... done (cluster.py:123, run_and_check) 2026-04-02 09:51:57 [ 363 ] DEBUG : Executing query SELECT * FROM dist on n2 (cluster.py:3602, query) 2026-04-02 09:51:58 [ 363 ] DEBUG : Executing query SELECT * FROM dist on n2 (cluster.py:3602, query) 2026-04-02 09:51:59 [ 363 ] DEBUG : Executing query SELECT errors_count FROM system.clusters WHERE cluster = 'replicas_cluster' AND host_name = 'n1' on n2 (cluster.py:3602, query) 2026-04-02 09:52:00 [ 363 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0_gw2/.env', '--project-name', 'roottestdistributedloadbalancing_gw2', '--file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0_gw2/n1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0_gw2/n2/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0_gw2/n3/docker-compose.yml', 'unpause', 'n1'] (cluster.py:113, run_and_check) 2026-04-02 09:52:04 [ 363 ] DEBUG : Stderr:Unpausing roottestdistributedloadbalancing_gw2_n1_1 ... (cluster.py:123, run_and_check) 2026-04-02 09:52:04 [ 363 ] DEBUG : Stderr:Unpausing roottestdistributedloadbalancing_gw2_n1_1 ... done (cluster.py:123, run_and_check) 2026-04-02 09:52:04 [ 363 ] DEBUG : Executing query SELECT * FROM dist on n2 (cluster.py:3602, query) 2026-04-02 09:52:07 [ 363 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n1 (cluster.py:3602, query) 2026-04-02 09:52:12 [ 363 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n2 (cluster.py:3602, query) 2026-04-02 09:52:15 [ 363 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n3 (cluster.py:3602, query) 2026-04-02 09:52:19 [ 363 ] DEBUG : Executing query SELECT hostName() FROM cluster(shards_cluster, system.query_log) WHERE initial_query_id = 'daa35579465944e38adb9aed942c2ff4' AND is_initial_query = 0 AND type = 'QueryFinish' ORDER BY event_date DESC, event_time DESC LIMIT 1 on n2 (cluster.py:3602, query) ============================== slowest durations =============================== 671.18s call test_globs_in_filepath/test.py::test_deep_structure 452.12s call test_distributed_load_balancing/test.py::test_load_balancing_first_or_random 355.27s call test_distributed_load_balancing/test.py::test_load_balancing_default 309.30s call test_distributed_load_balancing/test.py::test_load_balancing_nearest_hostname 306.25s call test_distributed_load_balancing/test.py::test_load_balancing_hostname_levenshtein_distance 296.20s call test_distributed_load_balancing/test.py::test_load_balancing_in_order 124.56s setup test_endpoint_macro_substitution/test.py::test_different_types 101.47s call test_http_handlers_config/test.py::test_defaults_http_handlers_config_order 93.84s call test_hedged_requests/test.py::test_async_query_sending 77.46s setup test_distributed_inter_server_secret/test.py::test_insecure 71.87s call test_hedged_requests/test.py::test_async_connect 69.83s setup test_encrypted_disk_replication/test.py::test_replication 67.17s call test_graphite_merge_tree/test.py::test_path_dangling_pointer 66.34s call test_hedged_requests/test.py::test_long_query 65.02s call test_hedged_requests/test.py::test_receive_timeout1 63.26s call test_hedged_requests/test.py::test_send_data2 62.26s call test_hedged_requests/test.py::test_send_data 59.93s call test_hedged_requests/test.py::test_send_table_status_sleep 59.87s call test_globs_in_filepath/test.py::test_linear_structure 59.84s call test_hedged_requests/test.py::test_stuck_replica 58.90s call test_hedged_requests/test.py::test_initial_receive_timeout 58.51s setup test_distributed_load_balancing/test.py::test_distributed_replica_max_ignored_errors 58.43s call test_distributed_insert_backward_compatibility/test.py::test_distributed_in_tuple 57.44s setup test_executable_user_defined_functions_config_reload/test.py::test 56.69s call test_hedged_requests/test.py::test_combination4 52.80s call test_hedged_requests/test.py::test_send_table_status_sleep2 52.68s call test_http_handlers_config/test.py::test_defaults_http_handlers 51.59s call test_http_handlers_config/test.py::test_config_static_handler 51.56s call test_http_handlers_config/test.py::test_predefined_query_handler 51.04s call test_hedged_requests/test.py::test_combination1 50.21s call test_http_handlers_config/test.py::test_relative_path_static_handler 49.53s setup test_hedged_requests/test.py::test_async_connect 49.17s call test_http_handlers_config/test.py::test_replicas_status_handler 48.84s call test_http_handlers_config/test.py::test_absolute_path_static_handler 47.50s call test_hedged_requests/test.py::test_receive_timeout2 44.98s call test_http_handlers_config/test.py::test_prometheus_handler 44.64s call test_http_handlers_config/test.py::test_headers_in_response 44.43s call test_http_handlers_config/test.py::test_dynamic_query_handler 43.57s call test_hot_reload_storage_policy/test.py::test_hot_reload_policy_distributed_table 40.35s teardown test_encrypted_disk_replication/test.py::test_zero_copy_replication 39.26s call test_hedged_requests/test.py::test_combination2 36.03s call test_hedged_requests/test.py::test_combination3 35.85s teardown test_endpoint_macro_substitution/test.py::test_select_by_type 35.81s setup test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_corrupted_big[0] 34.61s setup test_distributed_insert_backward_compatibility/test.py::test_distributed_in_tuple 34.48s setup test_dotnet_client/test.py::test_dotnet_client 33.96s call test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority] 33.78s call test_distributed_load_balancing/test.py::test_load_balancing_round_robin 33.68s call test_http_handlers_config/test.py::test_fixed_static_handler 33.34s setup test_graphite_merge_tree/test.py::test_broken_partial_rollup 31.59s call test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_corrupted_big[0] 31.06s setup test_globs_in_filepath/test.py::test_deep_structure 30.66s setup test_input_format_parallel_parsing_memory_tracking/test.py::test_memory_tracking_total 30.21s setup test_distributed_format/test.py::test_single_file[test_cluster_internal_replication] 29.12s call test_distributed_load_balancing/test.py::test_distributed_replica_max_ignored_errors 29.11s call test_input_format_parallel_parsing_memory_tracking/test.py::test_memory_tracking_total 28.99s call test_executable_user_defined_functions_config_reload/test.py::test 28.26s setup test_grpc_protocol_ssl/test.py::test_insecure_channel 27.91s call test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority_negative] 26.31s setup test_hot_reload_storage_policy/test.py::test_hot_reload_policy 25.82s teardown test_globs_in_filepath/test.py::test_table_function_and_virtual_columns 23.46s teardown test_hedged_requests/test.py::test_stuck_replica 22.74s call test_dotnet_client/test.py::test_dotnet_client 22.59s teardown test_grpc_protocol_ssl/test.py::test_wrong_client_certificate 20.11s teardown test_distributed_insert_backward_compatibility/test.py::test_distributed_in_tuple 18.08s teardown test_graphite_merge_tree/test.py::test_wrong_rollup_config 17.91s teardown test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_corrupted_big[0] 17.19s teardown test_dotnet_client/test.py::test_dotnet_client 16.68s teardown test_distributed_inter_server_secret/test.py::test_user_secure_cluster[pass-foo] 16.55s call test_globs_in_filepath/test.py::test_strange_filenames 16.47s call test_distributed_inter_server_secret/test.py::test_secure_insert_buffer_async 14.91s call test_globs_in_filepath/test.py::test_table_function_and_virtual_columns 14.33s call test_encrypted_disk_replication/test.py::test_replication 13.83s teardown test_executable_user_defined_functions_config_reload/test.py::test 13.75s teardown test_distributed_load_balancing/test.py::test_load_balancing_round_robin 13.47s teardown test_distributed_format/test.py::test_two_files[test_cluster_no_internal_replication] 12.99s call test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[pass-foo] 12.82s call test_distributed_inter_server_secret/test.py::test_user_secure_cluster[pass-foo] 11.95s call test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[default-] 11.79s call test_distributed_format/test.py::test_two_files[test_cluster_internal_replication] 11.24s call test_distributed_format/test.py::test_single_file_old[test_cluster_internal_replication] 11.01s teardown test_hot_reload_storage_policy/test.py::test_hot_reload_policy_distributed_table 10.89s call test_distributed_format/test.py::test_two_files[test_cluster_no_internal_replication] 10.60s call test_hot_reload_storage_policy/test.py::test_hot_reload_policy 10.40s call test_encrypted_disk_replication/test.py::test_zero_copy_replication 10.29s call test_distributed_inter_server_secret/test.py::test_user_secure_cluster[default-] 9.95s call test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_insecure_cluster[default-] 9.74s call test_distributed_format/test.py::test_single_file[test_cluster_no_internal_replication] 9.69s teardown test_input_format_parallel_parsing_memory_tracking/test.py::test_memory_tracking_total 9.50s call test_distributed_inter_server_secret/test.py::test_user_secure_cluster[nopass-] 9.26s call test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[nopass-] 8.66s call test_distributed_inter_server_secret/test.py::test_secure_cluster_distributed_over_distributed_different_users_cluster 8.18s call test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[nopass-] 7.84s call test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[nopass-] 7.70s call test_distributed_format/test.py::test_single_file[test_cluster_internal_replication] 7.51s call test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[default-] 7.31s call test_graphite_merge_tree/test.py::test_combined_rules_with_default 7.04s call test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[pass-foo] 6.83s call test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[default-] 6.80s call test_distributed_format/test.py::test_single_file_old[test_cluster_no_internal_replication] 6.57s call test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_secure_cluster[nopass-] 6.46s call test_graphite_merge_tree/test.py::test_rollup_versions 6.37s call test_endpoint_macro_substitution/test.py::test_select_by_type 5.98s call test_distributed_inter_server_secret/test.py::test_secure_disagree_insert 5.92s call test_graphite_merge_tree/test.py::test_combined_rules 5.91s call test_distributed_inter_server_secret/test.py::test_insecure_insert_async 5.77s call test_distributed_inter_server_secret/test.py::test_secure_insert_async 5.72s call test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_insecure_cluster[nopass-] 5.47s call test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_secure_cluster[default-] 5.38s call test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[pass-foo] 5.33s call test_graphite_merge_tree/test.py::test_system_graphite_retentions 4.92s call test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_insecure_cluster[pass-foo] 4.75s call test_distributed_inter_server_secret/test.py::test_secure_cluster_distributed_over_distributed_different_users_remote 4.68s setup test_distributed_load_balancing/test.py::test_load_balancing_hostname_levenshtein_distance 4.45s call test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_secure_cluster[pass-foo] 4.43s call test_distributed_inter_server_secret/test.py::test_secure_insert_sync 4.03s call test_distributed_inter_server_secret/test.py::test_insecure_insert_sync 3.95s call test_graphite_merge_tree/test.py::test_multiple_output_blocks 3.89s call test_graphite_merge_tree/test.py::test_rollup_aggregation 3.78s call test_graphite_merge_tree/test.py::test_broken_partial_rollup 3.72s setup test_distributed_load_balancing/test.py::test_load_balancing_default 3.69s setup test_distributed_load_balancing/test.py::test_load_balancing_round_robin 3.33s setup test_distributed_load_balancing/test.py::test_load_balancing_first_or_random 2.99s teardown test_encrypted_disk_replication/test.py::test_replication 2.66s setup test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority_negative] 2.61s setup test_distributed_load_balancing/test.py::test_load_balancing_in_order 2.40s setup test_distributed_load_balancing/test.py::test_load_balancing_nearest_hostname 2.38s call test_graphite_merge_tree/test.py::test_paths_not_matching_any_pattern 2.22s setup test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority] 2.20s call test_grpc_protocol_ssl/test.py::test_insecure_channel 2.01s call test_grpc_protocol_ssl/test.py::test_wrong_client_certificate 1.82s call test_graphite_merge_tree/test.py::test_rollup_aggregation_2 1.76s setup test_graphite_merge_tree/test.py::test_paths_not_matching_any_pattern 1.72s call test_endpoint_macro_substitution/test.py::test_different_types 1.61s teardown test_graphite_merge_tree/test.py::test_combined_rules_with_default 1.59s setup test_graphite_merge_tree/test.py::test_multiple_output_blocks 1.55s call test_graphite_merge_tree/test.py::test_multiple_paths_and_versions 1.37s call test_distributed_inter_server_secret/test.py::test_secure_disagree 1.37s setup test_graphite_merge_tree/test.py::test_system_graphite_retentions 1.37s call test_distributed_inter_server_secret/test.py::test_insecure 1.35s teardown test_graphite_merge_tree/test.py::test_rollup_aggregation 1.33s teardown test_graphite_merge_tree/test.py::test_combined_rules 1.31s teardown test_graphite_merge_tree/test.py::test_rollup_versions 1.29s setup test_graphite_merge_tree/test.py::test_combined_rules_with_default 1.28s setup test_graphite_merge_tree/test.py::test_rollup_versions 1.24s setup test_graphite_merge_tree/test.py::test_path_dangling_pointer 1.20s teardown test_graphite_merge_tree/test.py::test_system_graphite_retentions 1.20s setup test_graphite_merge_tree/test.py::test_multiple_paths_and_versions 1.20s setup test_graphite_merge_tree/test.py::test_combined_rules 1.16s teardown test_graphite_merge_tree/test.py::test_paths_not_matching_any_pattern 1.07s teardown test_graphite_merge_tree/test.py::test_rollup_aggregation_2 1.07s setup test_graphite_merge_tree/test.py::test_rollup_aggregation 1.06s teardown test_graphite_merge_tree/test.py::test_path_dangling_pointer 1.00s setup test_graphite_merge_tree/test.py::test_wrong_rollup_config 0.95s teardown test_graphite_merge_tree/test.py::test_broken_partial_rollup 0.93s call test_graphite_merge_tree/test.py::test_wrong_rollup_config 0.83s setup test_graphite_merge_tree/test.py::test_rollup_aggregation_2 0.72s teardown test_graphite_merge_tree/test.py::test_multiple_output_blocks 0.71s teardown test_graphite_merge_tree/test.py::test_multiple_paths_and_versions 0.67s call test_distributed_inter_server_secret/test.py::test_secure 0.61s call test_grpc_protocol_ssl/test.py::test_secure_channel 0.31s setup test_http_handlers_config/test.py::test_absolute_path_static_handler 0.06s teardown test_endpoint_macro_substitution/test.py::test_different_types 0.06s teardown test_hedged_requests/test.py::test_initial_receive_timeout 0.04s teardown test_globs_in_filepath/test.py::test_deep_structure 0.02s teardown test_http_handlers_config/test.py::test_absolute_path_static_handler 0.02s setup test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_insecure_cluster[default-] 0.02s teardown test_http_handlers_config/test.py::test_prometheus_handler 0.02s setup test_hedged_requests/test.py::test_send_table_status_sleep2 0.02s setup test_http_handlers_config/test.py::test_prometheus_handler 0.02s setup test_hedged_requests/test.py::test_combination4 0.02s teardown test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[default-] 0.02s setup test_distributed_inter_server_secret/test.py::test_insecure_insert_async 0.02s setup test_hedged_requests/test.py::test_combination2 0.01s setup test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[default-] 0.01s setup test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_insecure_cluster[nopass-] 0.01s setup test_globs_in_filepath/test.py::test_linear_structure 0.01s teardown test_globs_in_filepath/test.py::test_strange_filenames 0.01s teardown test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority] 0.01s setup test_hedged_requests/test.py::test_long_query 0.01s teardown test_hedged_requests/test.py::test_combination3 0.01s setup test_distributed_format/test.py::test_single_file[test_cluster_no_internal_replication] 0.01s teardown test_distributed_inter_server_secret/test.py::test_user_secure_cluster[nopass-] 0.01s teardown test_distributed_inter_server_secret/test.py::test_secure_insert_async 0.01s teardown test_distributed_inter_server_secret/test.py::test_secure_cluster_distributed_over_distributed_different_users_cluster 0.01s setup test_distributed_inter_server_secret/test.py::test_user_secure_cluster[default-] 0.01s teardown test_distributed_inter_server_secret/test.py::test_secure 0.01s teardown test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority_negative] 0.01s setup test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[pass-foo] 0.01s teardown test_hedged_requests/test.py::test_send_table_status_sleep2 0.01s setup test_http_handlers_config/test.py::test_predefined_query_handler 0.01s teardown test_http_handlers_config/test.py::test_defaults_http_handlers 0.01s setup test_hedged_requests/test.py::test_send_data2 0.01s teardown test_hedged_requests/test.py::test_send_table_status_sleep 0.01s setup test_http_handlers_config/test.py::test_defaults_http_handlers 0.01s setup test_distributed_inter_server_secret/test.py::test_secure_disagree 0.01s teardown test_distributed_load_balancing/test.py::test_load_balancing_hostname_levenshtein_distance 0.01s teardown test_http_handlers_config/test.py::test_relative_path_static_handler 0.01s setup test_http_handlers_config/test.py::test_relative_path_static_handler 0.01s teardown test_distributed_inter_server_secret/test.py::test_user_secure_cluster[default-] 0.01s teardown test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_insecure_cluster[pass-foo] 0.01s teardown test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_secure_cluster[default-] 0.01s setup test_http_handlers_config/test.py::test_fixed_static_handler 0.01s teardown test_http_handlers_config/test.py::test_predefined_query_handler 0.01s teardown test_http_handlers_config/test.py::test_fixed_static_handler 0.01s setup test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[nopass-] 0.01s teardown test_hedged_requests/test.py::test_combination1 0.01s setup test_distributed_inter_server_secret/test.py::test_secure_insert_async 0.01s setup test_endpoint_macro_substitution/test.py::test_select_by_type 0.01s teardown test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_insecure_cluster[default-] 0.01s setup test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_secure_cluster[default-] 0.01s setup test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[pass-foo] 0.01s setup test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_secure_cluster[pass-foo] 0.01s setup test_distributed_format/test.py::test_two_files[test_cluster_no_internal_replication] 0.00s setup test_hedged_requests/test.py::test_receive_timeout2 0.00s teardown test_hedged_requests/test.py::test_async_connect 0.00s teardown test_http_handlers_config/test.py::test_replicas_status_handler 0.00s teardown test_distributed_load_balancing/test.py::test_load_balancing_in_order 0.00s teardown test_distributed_inter_server_secret/test.py::test_insecure 0.00s setup test_hedged_requests/test.py::test_send_data 0.00s teardown test_hedged_requests/test.py::test_send_data 0.00s teardown test_hedged_requests/test.py::test_receive_timeout2 0.00s teardown test_distributed_load_balancing/test.py::test_load_balancing_first_or_random 0.00s setup test_distributed_inter_server_secret/test.py::test_user_secure_cluster[nopass-] 0.00s teardown test_hedged_requests/test.py::test_send_data2 0.00s teardown test_http_handlers_config/test.py::test_defaults_http_handlers_config_order 0.00s setup test_distributed_inter_server_secret/test.py::test_user_secure_cluster[pass-foo] 0.00s teardown test_hedged_requests/test.py::test_combination4 0.00s setup test_http_handlers_config/test.py::test_config_static_handler 0.00s setup test_hedged_requests/test.py::test_send_table_status_sleep 0.00s teardown test_distributed_load_balancing/test.py::test_load_balancing_default 0.00s setup test_http_handlers_config/test.py::test_headers_in_response 0.00s setup test_distributed_inter_server_secret/test.py::test_insecure_insert_sync 0.00s teardown test_distributed_inter_server_secret/test.py::test_secure_insert_buffer_async 0.00s setup test_hedged_requests/test.py::test_receive_timeout1 0.00s teardown test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_insecure_cluster[nopass-] 0.00s setup test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[default-] 0.00s setup test_hedged_requests/test.py::test_initial_receive_timeout 0.00s setup test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_insecure_cluster[pass-foo] 0.00s setup test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[nopass-] 0.00s setup test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_secure_cluster[nopass-] 0.00s setup test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[nopass-] 0.00s teardown test_distributed_load_balancing/test.py::test_distributed_replica_max_ignored_errors 0.00s setup test_hedged_requests/test.py::test_combination1 0.00s teardown test_hedged_requests/test.py::test_async_query_sending 0.00s setup test_distributed_format/test.py::test_single_file_old[test_cluster_internal_replication] 0.00s setup test_hedged_requests/test.py::test_async_query_sending 0.00s setup test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[pass-foo] 0.00s setup test_distributed_format/test.py::test_two_files[test_cluster_internal_replication] 0.00s setup test_distributed_inter_server_secret/test.py::test_secure_cluster_distributed_over_distributed_different_users_remote 0.00s teardown test_http_handlers_config/test.py::test_dynamic_query_handler 0.00s teardown test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[default-] 0.00s setup test_http_handlers_config/test.py::test_dynamic_query_handler 0.00s teardown test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[pass-foo] 0.00s setup test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[default-] 0.00s teardown test_distributed_inter_server_secret/test.py::test_secure_disagree_insert 0.00s setup test_hedged_requests/test.py::test_stuck_replica 0.00s setup test_encrypted_disk_replication/test.py::test_zero_copy_replication 0.00s setup test_http_handlers_config/test.py::test_replicas_status_handler 0.00s setup test_globs_in_filepath/test.py::test_table_function_and_virtual_columns 0.00s setup test_distributed_inter_server_secret/test.py::test_secure_insert_sync 0.00s teardown test_http_handlers_config/test.py::test_headers_in_response 0.00s setup test_distributed_inter_server_secret/test.py::test_secure_insert_buffer_async 0.00s teardown test_distributed_format/test.py::test_single_file[test_cluster_internal_replication] 0.00s teardown test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[pass-foo] 0.00s teardown test_hedged_requests/test.py::test_receive_timeout1 0.00s teardown test_hedged_requests/test.py::test_long_query 0.00s teardown test_globs_in_filepath/test.py::test_linear_structure 0.00s setup test_grpc_protocol_ssl/test.py::test_secure_channel 0.00s setup test_globs_in_filepath/test.py::test_strange_filenames 0.00s setup test_distributed_inter_server_secret/test.py::test_secure 0.00s setup test_hot_reload_storage_policy/test.py::test_hot_reload_policy_distributed_table 0.00s setup test_distributed_format/test.py::test_single_file_old[test_cluster_no_internal_replication] 0.00s setup test_distributed_inter_server_secret/test.py::test_secure_disagree_insert 0.00s setup test_hedged_requests/test.py::test_combination3 0.00s teardown test_http_handlers_config/test.py::test_config_static_handler 0.00s teardown test_distributed_load_balancing/test.py::test_load_balancing_nearest_hostname 0.00s setup test_grpc_protocol_ssl/test.py::test_wrong_client_certificate 0.00s teardown test_distributed_inter_server_secret/test.py::test_insecure_insert_sync 0.00s teardown test_grpc_protocol_ssl/test.py::test_insecure_channel 0.00s setup test_http_handlers_config/test.py::test_defaults_http_handlers_config_order 0.00s teardown test_hot_reload_storage_policy/test.py::test_hot_reload_policy 0.00s setup test_distributed_inter_server_secret/test.py::test_secure_cluster_distributed_over_distributed_different_users_cluster 0.00s teardown test_hedged_requests/test.py::test_combination2 0.00s teardown test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[nopass-] 0.00s teardown test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_secure_cluster[pass-foo] 0.00s teardown test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[nopass-] 0.00s teardown test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[nopass-] 0.00s teardown test_distributed_format/test.py::test_single_file[test_cluster_no_internal_replication] 0.00s teardown test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[default-] 0.00s teardown test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_secure_cluster[nopass-] 0.00s teardown test_distributed_format/test.py::test_two_files[test_cluster_internal_replication] 0.00s teardown test_distributed_inter_server_secret/test.py::test_insecure_insert_async 0.00s teardown test_distributed_inter_server_secret/test.py::test_secure_disagree 0.00s teardown test_distributed_format/test.py::test_single_file_old[test_cluster_internal_replication] 0.00s teardown test_distributed_inter_server_secret/test.py::test_secure_insert_sync 0.00s teardown test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[pass-foo] 0.00s teardown test_distributed_inter_server_secret/test.py::test_secure_cluster_distributed_over_distributed_different_users_remote 0.00s teardown test_distributed_format/test.py::test_single_file_old[test_cluster_no_internal_replication] 0.00s teardown test_grpc_protocol_ssl/test.py::test_secure_channel =========================== short test summary info ============================ FAILED test_distributed_load_balancing/test.py::test_distributed_replica_max_ignored_errors PASSED test_grpc_protocol_ssl/test.py::test_insecure_channel PASSED test_grpc_protocol_ssl/test.py::test_secure_channel PASSED test_grpc_protocol_ssl/test.py::test_wrong_client_certificate PASSED test_graphite_merge_tree/test.py::test_broken_partial_rollup PASSED test_distributed_format/test.py::test_single_file[test_cluster_internal_replication] PASSED test_graphite_merge_tree/test.py::test_combined_rules PASSED test_distributed_format/test.py::test_single_file[test_cluster_no_internal_replication] PASSED test_http_handlers_config/test.py::test_absolute_path_static_handler PASSED test_graphite_merge_tree/test.py::test_combined_rules_with_default PASSED test_distributed_format/test.py::test_single_file_old[test_cluster_internal_replication] PASSED test_graphite_merge_tree/test.py::test_multiple_output_blocks PASSED test_distributed_format/test.py::test_single_file_old[test_cluster_no_internal_replication] PASSED test_graphite_merge_tree/test.py::test_multiple_paths_and_versions PASSED test_distributed_format/test.py::test_two_files[test_cluster_internal_replication] PASSED test_distributed_inter_server_secret/test.py::test_insecure PASSED test_encrypted_disk_replication/test.py::test_replication PASSED test_distributed_inter_server_secret/test.py::test_insecure_insert_async PASSED test_distributed_format/test.py::test_two_files[test_cluster_no_internal_replication] PASSED test_distributed_inter_server_secret/test.py::test_insecure_insert_sync PASSED test_encrypted_disk_replication/test.py::test_zero_copy_replication PASSED test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_insecure_cluster[default-] PASSED test_http_handlers_config/test.py::test_config_static_handler PASSED test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_insecure_cluster[nopass-] PASSED test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_insecure_cluster[pass-foo] PASSED test_dotnet_client/test.py::test_dotnet_client PASSED test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_secure_cluster[default-] PASSED test_hedged_requests/test.py::test_async_connect PASSED test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_secure_cluster[nopass-] PASSED test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_secure_cluster[pass-foo] PASSED test_endpoint_macro_substitution/test.py::test_different_types PASSED test_endpoint_macro_substitution/test.py::test_select_by_type PASSED test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[default-] PASSED test_graphite_merge_tree/test.py::test_path_dangling_pointer PASSED test_graphite_merge_tree/test.py::test_paths_not_matching_any_pattern PASSED test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[nopass-] PASSED test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[pass-foo] PASSED test_graphite_merge_tree/test.py::test_rollup_aggregation PASSED test_graphite_merge_tree/test.py::test_rollup_aggregation_2 PASSED test_http_handlers_config/test.py::test_defaults_http_handlers PASSED test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[default-] PASSED test_graphite_merge_tree/test.py::test_rollup_versions PASSED test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[nopass-] PASSED test_graphite_merge_tree/test.py::test_system_graphite_retentions PASSED test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[pass-foo] PASSED test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_corrupted_big[0] PASSED test_distributed_inter_server_secret/test.py::test_secure PASSED test_graphite_merge_tree/test.py::test_wrong_rollup_config PASSED test_hot_reload_storage_policy/test.py::test_hot_reload_policy PASSED test_distributed_inter_server_secret/test.py::test_secure_cluster_distributed_over_distributed_different_users_cluster PASSED test_distributed_inter_server_secret/test.py::test_secure_cluster_distributed_over_distributed_different_users_remote PASSED test_distributed_inter_server_secret/test.py::test_secure_disagree PASSED test_distributed_inter_server_secret/test.py::test_secure_disagree_insert PASSED test_distributed_inter_server_secret/test.py::test_secure_insert_async PASSED test_distributed_inter_server_secret/test.py::test_secure_insert_buffer_async PASSED test_hedged_requests/test.py::test_async_query_sending PASSED test_executable_user_defined_functions_config_reload/test.py::test PASSED test_distributed_inter_server_secret/test.py::test_secure_insert_sync PASSED test_hot_reload_storage_policy/test.py::test_hot_reload_policy_distributed_table PASSED test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[default-] PASSED test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[nopass-] PASSED test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[pass-foo] PASSED test_http_handlers_config/test.py::test_defaults_http_handlers_config_order PASSED test_distributed_insert_backward_compatibility/test.py::test_distributed_in_tuple PASSED test_distributed_inter_server_secret/test.py::test_user_secure_cluster[default-] PASSED test_hedged_requests/test.py::test_combination1 PASSED test_distributed_inter_server_secret/test.py::test_user_secure_cluster[nopass-] PASSED test_distributed_inter_server_secret/test.py::test_user_secure_cluster[pass-foo] PASSED test_input_format_parallel_parsing_memory_tracking/test.py::test_memory_tracking_total PASSED test_http_handlers_config/test.py::test_dynamic_query_handler PASSED test_hedged_requests/test.py::test_combination2 PASSED test_http_handlers_config/test.py::test_fixed_static_handler PASSED test_hedged_requests/test.py::test_combination3 PASSED test_http_handlers_config/test.py::test_headers_in_response PASSED test_hedged_requests/test.py::test_combination4 PASSED test_http_handlers_config/test.py::test_predefined_query_handler PASSED test_distributed_load_balancing/test.py::test_load_balancing_default PASSED test_hedged_requests/test.py::test_initial_receive_timeout PASSED test_http_handlers_config/test.py::test_prometheus_handler PASSED test_hedged_requests/test.py::test_long_query PASSED test_http_handlers_config/test.py::test_relative_path_static_handler PASSED test_http_handlers_config/test.py::test_replicas_status_handler PASSED test_hedged_requests/test.py::test_receive_timeout1 PASSED test_hedged_requests/test.py::test_receive_timeout2 PASSED test_hedged_requests/test.py::test_send_data PASSED test_globs_in_filepath/test.py::test_deep_structure PASSED test_hedged_requests/test.py::test_send_data2 PASSED test_globs_in_filepath/test.py::test_linear_structure PASSED test_globs_in_filepath/test.py::test_strange_filenames PASSED test_globs_in_filepath/test.py::test_table_function_and_virtual_columns PASSED test_hedged_requests/test.py::test_send_table_status_sleep PASSED test_hedged_requests/test.py::test_send_table_status_sleep2 PASSED test_distributed_load_balancing/test.py::test_load_balancing_first_or_random PASSED test_hedged_requests/test.py::test_stuck_replica PASSED test_distributed_load_balancing/test.py::test_load_balancing_hostname_levenshtein_distance PASSED test_distributed_load_balancing/test.py::test_load_balancing_in_order PASSED test_distributed_load_balancing/test.py::test_load_balancing_nearest_hostname PASSED test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority] PASSED test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority_negative] PASSED test_distributed_load_balancing/test.py::test_load_balancing_round_robin ================== 1 failed, 99 passed in 1947.94s (0:32:27) =================== Traceback (most recent call last): File "/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration/./runner", line 445, in subprocess.check_call(cmd, shell=True) File "/usr/lib/python3.10/subprocess.py", line 369, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command 'docker run --rm --name clickhouse_integration_tests_v1snvh --privileged --dns-search='.' --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-odbc-bridge:/clickhouse-odbc-bridge --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-library-bridge:/clickhouse-library-bridge --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/backupview:/ClickHouse/utils/backupview --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/grpc-client/pb2:/ClickHouse/utils/grpc-client/pb2 --volume=/run:/run/host:ro --volume=clickhouse_integration_tests_volume:/var/lib/docker -e DOCKER_DOTNET_CLIENT_TAG=11de0b29a15d -e DOCKER_HELPER_TAG=2cffe1eae894 -e DOCKER_BASE_TAG=1e0b53d756cf -e DOCKER_KERBEROS_KDC_TAG=9391ecdee8d7 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=9bec2a638e6e -e DOCKER_MYSQL_JAVA_CLIENT_TAG=766bff31cfe4 -e DOCKER_MYSQL_JS_CLIENT_TAG=41ba7c2ec2a1 -e DOCKER_MYSQL_PHP_CLIENT_TAG=88be89c1e3b6 -e DOCKER_NGINX_DAV_TAG=b55ac9cd7519 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=a4eff5c7f4d6 -e DOCKER_PYTHON_BOTTLE_TAG=caad4729259e -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e CLICKHOUSE_USE_OLD_ANALYZER=1 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="--dist=loadfile -n 10 -rfEps --run-id=0 --color=no --durations=0 'test_distributed_format/test.py::test_single_file[test_cluster_internal_replication]' 'test_distributed_format/test.py::test_single_file[test_cluster_no_internal_replication]' 'test_distributed_format/test.py::test_single_file_old[test_cluster_internal_replication]' 'test_distributed_format/test.py::test_single_file_old[test_cluster_no_internal_replication]' 'test_distributed_format/test.py::test_two_files[test_cluster_internal_replication]' 'test_distributed_format/test.py::test_two_files[test_cluster_no_internal_replication]' test_distributed_insert_backward_compatibility/test.py::test_distributed_in_tuple test_distributed_inter_server_secret/test.py::test_insecure test_distributed_inter_server_secret/test.py::test_insecure_insert_async test_distributed_inter_server_secret/test.py::test_insecure_insert_sync 'test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_insecure_cluster[default-]' 'test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_insecure_cluster[nopass-]' 'test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_insecure_cluster[pass-foo]' 'test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_secure_cluster[default-]' 'test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_secure_cluster[nopass-]' 'test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_secure_cluster[pass-foo]' 'test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[default-]' 'test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[nopass-]' 'test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[pass-foo]' 'test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[default-]' 'test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[nopass-]' 'test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[pass-foo]' test_distributed_inter_server_secret/test.py::test_secure test_distributed_inter_server_secret/test.py::test_secure_cluster_distributed_over_distributed_different_users_cluster test_distributed_inter_server_secret/test.py::test_secure_cluster_distributed_over_distributed_different_users_remote test_distributed_inter_server_secret/test.py::test_secure_disagree test_distributed_inter_server_secret/test.py::test_secure_disagree_insert test_distributed_inter_server_secret/test.py::test_secure_insert_async test_distributed_inter_server_secret/test.py::test_secure_insert_buffer_async test_distributed_inter_server_secret/test.py::test_secure_insert_sync 'test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[default-]' 'test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[nopass-]' 'test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[pass-foo]' 'test_distributed_inter_server_secret/test.py::test_user_secure_cluster[default-]' 'test_distributed_inter_server_secret/test.py::test_user_secure_cluster[nopass-]' 'test_distributed_inter_server_secret/test.py::test_user_secure_cluster[pass-foo]' test_distributed_load_balancing/test.py::test_distributed_replica_max_ignored_errors test_distributed_load_balancing/test.py::test_load_balancing_default test_distributed_load_balancing/test.py::test_load_balancing_first_or_random test_distributed_load_balancing/test.py::test_load_balancing_hostname_levenshtein_distance test_distributed_load_balancing/test.py::test_load_balancing_in_order test_distributed_load_balancing/test.py::test_load_balancing_nearest_hostname 'test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority]' 'test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority_negative]' test_distributed_load_balancing/test.py::test_load_balancing_round_robin test_dotnet_client/test.py::test_dotnet_client test_encrypted_disk_replication/test.py::test_replication test_encrypted_disk_replication/test.py::test_zero_copy_replication test_endpoint_macro_substitution/test.py::test_different_types test_endpoint_macro_substitution/test.py::test_select_by_type test_executable_user_defined_functions_config_reload/test.py::test test_globs_in_filepath/test.py::test_deep_structure test_globs_in_filepath/test.py::test_linear_structure test_globs_in_filepath/test.py::test_strange_filenames test_globs_in_filepath/test.py::test_table_function_and_virtual_columns test_graphite_merge_tree/test.py::test_broken_partial_rollup test_graphite_merge_tree/test.py::test_combined_rules test_graphite_merge_tree/test.py::test_combined_rules_with_default test_graphite_merge_tree/test.py::test_multiple_output_blocks test_graphite_merge_tree/test.py::test_multiple_paths_and_versions test_graphite_merge_tree/test.py::test_path_dangling_pointer test_graphite_merge_tree/test.py::test_paths_not_matching_any_pattern test_graphite_merge_tree/test.py::test_rollup_aggregation test_graphite_merge_tree/test.py::test_rollup_aggregation_2 test_graphite_merge_tree/test.py::test_rollup_versions test_graphite_merge_tree/test.py::test_system_graphite_retentions test_graphite_merge_tree/test.py::test_wrong_rollup_config test_grpc_protocol_ssl/test.py::test_insecure_channel test_grpc_protocol_ssl/test.py::test_secure_channel test_grpc_protocol_ssl/test.py::test_wrong_client_certificate test_hedged_requests/test.py::test_async_connect test_hedged_requests/test.py::test_async_query_sending test_hedged_requests/test.py::test_combination1 test_hedged_requests/test.py::test_combination2 test_hedged_requests/test.py::test_combination3 test_hedged_requests/test.py::test_combination4 test_hedged_requests/test.py::test_initial_receive_timeout test_hedged_requests/test.py::test_long_query test_hedged_requests/test.py::test_receive_timeout1 test_hedged_requests/test.py::test_receive_timeout2 test_hedged_requests/test.py::test_send_data test_hedged_requests/test.py::test_send_data2 test_hedged_requests/test.py::test_send_table_status_sleep test_hedged_requests/test.py::test_send_table_status_sleep2 test_hedged_requests/test.py::test_stuck_replica test_hot_reload_storage_policy/test.py::test_hot_reload_policy test_hot_reload_storage_policy/test.py::test_hot_reload_policy_distributed_table test_http_handlers_config/test.py::test_absolute_path_static_handler test_http_handlers_config/test.py::test_config_static_handler test_http_handlers_config/test.py::test_defaults_http_handlers test_http_handlers_config/test.py::test_defaults_http_handlers_config_order test_http_handlers_config/test.py::test_dynamic_query_handler test_http_handlers_config/test.py::test_fixed_static_handler test_http_handlers_config/test.py::test_headers_in_response test_http_handlers_config/test.py::test_predefined_query_handler test_http_handlers_config/test.py::test_prometheus_handler test_http_handlers_config/test.py::test_relative_path_static_handler test_http_handlers_config/test.py::test_replicas_status_handler test_input_format_parallel_parsing_memory_tracking/test.py::test_memory_tracking_total 'test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_corrupted_big[0]' -vvv" altinityinfra/integration-tests-runner:37a9815fd2fa ' returned non-zero exit status 1.